Variational Context: Exploiting Visual and Textual Context for Grounding Referring Expressions

Grounding Referring Expressions in Images by Variational Context

We focus on grounding (i.e., localizing or linking) referring expressions in images, e.g., “largest elephant standing behind baby elephant”. This is a general yet challenging vision-language task since it does not only require the localization of objects, but also the multimodal comprehension of context — visual attributes (e.g., “largest”, “baby”) and relationships (e.g., “behind”) that help t...

Modeling Context in Referring Expressions

Humans refer to objects in their environments all the time, especially in dialogue with other people. We explore generating and comprehending natural language referring expressions for objects in images. In particular, we focus on incorporating better measures of visual context into referring expression models and find that visual comparison to other objects within an image helps improve perfor...

Generating Referring Expressions in a Multimodal Context

In this paper an algorithm for the generation of referring expressions in a multimodal setting is presented. The algorithm is based on empirical studies of how humans refer to objects in a shared workspace. The main ingredients of the algorithm are the following. First, the addition of deictic pointing gestures, where the decision to point is determined by two factors: the effort of pointing (m...

Efficient Context-Sensitive Generation of Referring Expressions

3 A Modification of the Algorithm Based on Salience 5 3.1 Motivation: Determining the Context Set : : : : : : : : : : : : : 5 3.2 Preliminaries : : : : : : : : : : : : : : : : : : : : : : : : : : : : 6 3.3 Outline of the Modified Algorithm : : : : : : : : : : : : : : : : : 7 3.4 Examples : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 9 3.5 Discussion : : : : : : : : : : : : : : : :...

Efficient Context-Sensitive Generation of Referring Expressions